Skip to content

feat: add persisted sync metadata support#1380

Draft
samwillis wants to merge 12 commits intomainfrom
persisted-sync-metadata
Draft

feat: add persisted sync metadata support#1380
samwillis wants to merge 12 commits intomainfrom
persisted-sync-metadata

Conversation

@samwillis
Copy link
Collaborator

@samwillis samwillis commented Mar 17, 2026

Summary

  • add transactional persisted sync metadata support across @tanstack/db, db-sqlite-persisted-collection-core, query-db-collection, and electric-db-collection
  • persist row-scoped metadata with synced rows and collection-scoped metadata alongside persisted collection state
  • use that metadata to preserve query ownership/retention across restart and to persist Electric resume state

How To Use

Custom sync implementations

If a collection is running on a persistence layer that supports this feature, the sync function now receives an optional metadata API alongside begin, write, commit, markReady, and truncate.

sync: ({ begin, write, commit, metadata }) => {
  const startupValue = metadata?.collection.get('my-sync:startup')

  begin()
  write({
    type: 'update',
    value: row,
    metadata: { source: 'remote' },
  })
  metadata?.collection.set('my-sync:startup', { ready: true })
  commit()
}

Available operations:

  • metadata.row.get(key)
  • metadata.row.set(key, value)
  • metadata.row.delete(key)
  • metadata.collection.get(key)
  • metadata.collection.set(key, value)
  • metadata.collection.delete(key)
  • metadata.collection.list(prefix?)

Behavior:

  • startup reads through get / list are allowed before any transaction is opened
  • metadata writes are transactional and must happen inside begin() / commit()
  • write({ metadata }) and metadata.row.set() target the same row metadata store

SQLite persisted collections

persistedCollectionOptions(...) now persists both row metadata and collection metadata when used with db-sqlite-persisted-collection-core.

Typical usage looks like:

const collection = createCollection(
  persistedCollectionOptions({
    ...someCollectionOptions,
    persistence: {
      adapter: sqliteAdapter,
    },
  }),
)

What you get automatically:

  • row metadata hydrates with persisted rows
  • collection metadata is loaded during startup before persisted sync work begins
  • row data, row metadata, and collection metadata commit in the same persisted transaction

Query collections

Query collections can now retain persisted ownership metadata across restarts.

const collection = createCollection(
  persistedCollectionOptions({
    ...queryCollectionOptions({
      id: 'messages',
      queryClient,
      queryKey: ['messages', roomId],
      queryFn: fetchMessages,
      getKey: (row) => row.id,
      persistedGcTime: Number.POSITIVE_INFINITY,
    }),
    persistence: {
      adapter: sqliteAdapter,
    },
  }),
)

What this enables:

  • row ownership is persisted on rows, so warm starts do not incorrectly delete disjoint pre-hydrated rows
  • query retention is separate from in-memory TanStack Query gcTime
  • persistedGcTime can be finite or effectively indefinite for offline-first flows
  • when a retained query is requested again, reconciliation diffing uses that query's owned-row baseline instead of the whole collection

Electric collections

Electric collections now persist resume metadata at collection scope when wrapped with persisted collection options.

const collection = createCollection(
  persistedCollectionOptions({
    ...electricCollectionOptions({
      id: 'todos',
      getKey: (row) => row.id,
      shapeOptions: {
        url: electricUrl,
        params: { table: 'todos' },
      },
    }),
    persistence: {
      adapter: sqliteAdapter,
    },
  }),
)

What this enables:

  • persisted startup can reuse a saved Electric offset / handle resume point
  • resume metadata is committed transactionally with the batch that made it valid
  • must-refetch/reset paths write a reset marker before reloading so restart does not resume from stale stream state

Test plan

  • Run pnpm vitest --run packages/db/tests/collection.test.ts
  • Run pnpm --filter @tanstack/query-db-collection exec vitest run tests/query.test.ts --coverage.enabled false
  • Run pnpm --filter @tanstack/electric-db-collection exec vitest run tests/electric.test.ts --coverage.enabled false
  • Run pnpm --filter @tanstack/db-sqlite-persisted-collection-core exec vitest run tests/persisted.test.ts tests/sqlite-core-adapter.test.ts --coverage.enabled false

Notes

  • The persisted SQLite package tests are functionally green in package-local runs.
  • There is still noisy stderr in some query tests from expected error-path coverage and existing cleanupQueryIfIdle warnings.

@pkg-pr-new
Copy link

pkg-pr-new bot commented Mar 17, 2026

More templates

@tanstack/angular-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/angular-db@1380

@tanstack/db

npm i https://pkg.pr.new/TanStack/db/@tanstack/db@1380

@tanstack/db-browser-wa-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-browser-wa-sqlite-persisted-collection@1380

@tanstack/db-ivm

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-ivm@1380

@tanstack/db-react-native-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-react-native-sqlite-persisted-collection@1380

@tanstack/db-sqlite-persisted-collection-core

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-sqlite-persisted-collection-core@1380

@tanstack/electric-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/electric-db-collection@1380

@tanstack/offline-transactions

npm i https://pkg.pr.new/TanStack/db/@tanstack/offline-transactions@1380

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/powersync-db-collection@1380

@tanstack/query-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/query-db-collection@1380

@tanstack/react-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/react-db@1380

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/rxdb-db-collection@1380

@tanstack/solid-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/solid-db@1380

@tanstack/svelte-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/svelte-db@1380

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/trailbase-db-collection@1380

@tanstack/vue-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/vue-db@1380

commit: a1a2f2e

@github-actions
Copy link
Contributor

github-actions bot commented Mar 17, 2026

Size Change: +504 B (+0.46%)

Total Size: 111 kB

Filename Size Change
./packages/db/dist/esm/collection/state.js 5.26 kB +54 B (+1.04%)
./packages/db/dist/esm/collection/sync.js 2.88 kB +450 B (+18.5%) ⚠️
ℹ️ View Unchanged
Filename Size
./packages/db/dist/esm/collection/change-events.js 1.39 kB
./packages/db/dist/esm/collection/changes.js 1.38 kB
./packages/db/dist/esm/collection/cleanup-queue.js 810 B
./packages/db/dist/esm/collection/events.js 434 B
./packages/db/dist/esm/collection/index.js 3.69 kB
./packages/db/dist/esm/collection/indexes.js 2.35 kB
./packages/db/dist/esm/collection/lifecycle.js 1.76 kB
./packages/db/dist/esm/collection/mutations.js 2.47 kB
./packages/db/dist/esm/collection/subscription.js 3.71 kB
./packages/db/dist/esm/collection/transaction-metadata.js 144 B
./packages/db/dist/esm/deferred.js 207 B
./packages/db/dist/esm/errors.js 4.83 kB
./packages/db/dist/esm/event-emitter.js 748 B
./packages/db/dist/esm/index.js 2.85 kB
./packages/db/dist/esm/indexes/auto-index.js 777 B
./packages/db/dist/esm/indexes/base-index.js 766 B
./packages/db/dist/esm/indexes/btree-index.js 2.17 kB
./packages/db/dist/esm/indexes/lazy-index.js 1.24 kB
./packages/db/dist/esm/indexes/reverse-index.js 538 B
./packages/db/dist/esm/local-only.js 890 B
./packages/db/dist/esm/local-storage.js 2.1 kB
./packages/db/dist/esm/optimistic-action.js 359 B
./packages/db/dist/esm/paced-mutations.js 496 B
./packages/db/dist/esm/proxy.js 3.75 kB
./packages/db/dist/esm/query/builder/functions.js 792 B
./packages/db/dist/esm/query/builder/index.js 5.15 kB
./packages/db/dist/esm/query/builder/ref-proxy.js 1.05 kB
./packages/db/dist/esm/query/compiler/evaluators.js 1.62 kB
./packages/db/dist/esm/query/compiler/expressions.js 430 B
./packages/db/dist/esm/query/compiler/group-by.js 2.69 kB
./packages/db/dist/esm/query/compiler/index.js 3.62 kB
./packages/db/dist/esm/query/compiler/joins.js 2.11 kB
./packages/db/dist/esm/query/compiler/order-by.js 1.5 kB
./packages/db/dist/esm/query/compiler/select.js 1.11 kB
./packages/db/dist/esm/query/effect.js 4.78 kB
./packages/db/dist/esm/query/expression-helpers.js 1.43 kB
./packages/db/dist/esm/query/ir.js 784 B
./packages/db/dist/esm/query/live-query-collection.js 360 B
./packages/db/dist/esm/query/live/collection-config-builder.js 7.63 kB
./packages/db/dist/esm/query/live/collection-registry.js 264 B
./packages/db/dist/esm/query/live/collection-subscriber.js 1.94 kB
./packages/db/dist/esm/query/live/internal.js 145 B
./packages/db/dist/esm/query/live/utils.js 1.57 kB
./packages/db/dist/esm/query/optimizer.js 2.62 kB
./packages/db/dist/esm/query/predicate-utils.js 2.97 kB
./packages/db/dist/esm/query/query-once.js 359 B
./packages/db/dist/esm/query/subset-dedupe.js 960 B
./packages/db/dist/esm/scheduler.js 1.3 kB
./packages/db/dist/esm/SortedMap.js 1.3 kB
./packages/db/dist/esm/strategies/debounceStrategy.js 247 B
./packages/db/dist/esm/strategies/queueStrategy.js 428 B
./packages/db/dist/esm/strategies/throttleStrategy.js 246 B
./packages/db/dist/esm/transactions.js 2.9 kB
./packages/db/dist/esm/utils.js 927 B
./packages/db/dist/esm/utils/browser-polyfills.js 304 B
./packages/db/dist/esm/utils/btree.js 5.61 kB
./packages/db/dist/esm/utils/comparison.js 1.05 kB
./packages/db/dist/esm/utils/cursor.js 457 B
./packages/db/dist/esm/utils/index-optimization.js 1.54 kB
./packages/db/dist/esm/utils/type-guards.js 157 B
./packages/db/dist/esm/virtual-props.js 360 B

compressed-size-action::db-package-size

@github-actions
Copy link
Contributor

github-actions bot commented Mar 17, 2026

Size Change: 0 B

Total Size: 4.23 kB

ℹ️ View Unchanged
Filename Size
./packages/react-db/dist/esm/index.js 249 B
./packages/react-db/dist/esm/useLiveInfiniteQuery.js 1.32 kB
./packages/react-db/dist/esm/useLiveQuery.js 1.34 kB
./packages/react-db/dist/esm/useLiveQueryEffect.js 355 B
./packages/react-db/dist/esm/useLiveSuspenseQuery.js 559 B
./packages/react-db/dist/esm/usePacedMutations.js 401 B

compressed-size-action::react-db-package-size

samwillis and others added 6 commits March 17, 2026 16:37
Document the transactional metadata model for persisted collections, including row and collection metadata, query retention, and Electric resume state.

Made-with: Cursor
Break the persisted sync metadata RFC into phased implementation docs covering the core API, SQLite integration, query collection, Electric collection, and required invariants tests.

Made-with: Cursor
Tighten the RFC and phased plan around startup metadata reads, query-owned reconciliation, cold-row retention cleanup, replay fallback behavior, and Electric reset semantics.

Made-with: Cursor
Add transactional row and collection metadata plumbing across core sync state, SQLite persistence, query collections, and Electric resume state so persisted ownership and resume metadata survive restarts.

Made-with: Cursor
Buffer persisted metadata writes within wrapper transactions and dedupe concurrent collection setup so warm starts no longer trip missing sync transaction errors or collection registry races.

Made-with: Cursor
@samwillis samwillis force-pushed the persisted-sync-metadata branch from fce939f to 7591ee1 Compare March 17, 2026 16:37
samwillis and others added 6 commits March 17, 2026 17:40
Drop the RFC and phased implementation plan from the branch while leaving the local working copies in place.

Made-with: Cursor
Finish the core persisted metadata follow-through so reloads, retained query ownership, and Electric resume/reset state behave correctly across startup and recovery while clarifying metadata semantics around inserts and cleanup.

Made-with: Cursor
Finish the remaining persisted metadata work by adding cold-row retained query cleanup, runtime TTL expiry, stronger Electric resume identity checks, and metadata delta replay for follower recovery while keeping reload fallback for reset-like cases.

Made-with: Cursor
Use the persisted JSON encoder for replay payloads so bigint and date values survive applied_tx serialization and package-level SQLite adapter tests pass under the CLI runtime.

Made-with: Cursor
Copy link
Contributor

@kevin-dp kevin-dp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness: Does It Fix the Bug?

Yes — the core fix is sound. The key change in query.ts is that applySuccessfulResult now computes previouslyOwnedRows from persisted ownership metadata rather than iterating collection._state.syncedData.entries():

const previouslyOwnedRows = shouldUsePersistedBaseline
  ? new Set(persistedBaseline.keys())
  : getHydratedOwnedRowsForQueryBaseline(hashedQueryKey)

This means an empty live query only removes rows it previously owned — not rows belonging to history. The ownership metadata (queryCollection.owners) is persisted alongside rows so it survives restarts.


Issues Found

MEDIUM Severity

1. markReady() error handling removed (persisted.ts)

The old code had a .catch() on runtime.ensureStarted() that called params.markReady() on failure:

// Old:
void runtime.ensureStarted()
  .then(() => { params.markReady() })
  .catch((error) => {
    console.warn(`Failed persisted sync startup before markReady:`, error)
    params.markReady()
  })

// New:
void (fullStartPromise ?? runtime.ensureStarted()).then(() => {
  params.markReady()
})

If ensureStarted() rejects (e.g., SQLite driver error during hydration or index bootstrap), markReady() is never called, leaving the collection stuck in a permanent loading state. The old code correctly degraded gracefully by marking ready anyway.

2. truncate() preserves collectionMetadataWrites but clears rowMetadataWrites — undocumented asymmetry

In both the core sync.ts and the persisted wrapper createWrappedSyncConfig, the truncate handler clears row-level state but preserves collection-level metadata writes:

openTransaction.operations = []
openTransaction.rowMetadataWrites.clear()
openTransaction.truncate = true
// collectionMetadataWrites NOT cleared

This is intentional — the Electric resume metadata writes happen before truncate and must survive — and tests verify it explicitly. But the asymmetry is subtle and load-bearing. A comment explaining why collection metadata survives truncate would prevent future contributors from "fixing" this.

LOW Severity

3. ALTER TABLE ... ADD COLUMN migration wrapped in bare try {} catch {}

try {
  await this.driver.exec(`ALTER TABLE applied_tx ADD COLUMN replay_json TEXT`)
} catch {}

This is a standard SQLite migration pattern (SQLite lacks ALTER TABLE ADD COLUMN IF NOT EXISTS), but the empty catch swallows all errors — not just "column already exists." If the driver has a permissions issue or the table is corrupt, the error is lost and the failure would surface later as a confusing "column not found" error.

4. getStableShapeIdentity uses JSON.stringify for identity comparison

function getStableShapeIdentity(shapeOptions) {
  return JSON.stringify({ url: shapeOptions.url, params: shapeOptions.params ?? null })
}

JSON.stringify iterates own enumerable properties in insertion order, so { table: 'x', where: 'y' } and { where: 'y', table: 'x' } produce different strings. If a user's params object is constructed with different key ordering between sessions, the persisted resume identity would mismatch and trigger a needless fresh sync. The codebase already has stableSerialize in the persisted package that handles this — consider using it here.

5. cancelledLoads uses WeakSet<object> with reference identity

const cancelledLoads = new WeakSet<object>()
// In loadSubset:
cancelledLoads.delete(options as object)
// In unloadSubset:
cancelledLoads.add(options as object)

This only works if the exact same options object reference is passed to both loadSubset and unloadSubset. The collection's internal subscription management likely does pass the same reference, but the correctness depends on an undocumented invariant of the caller.


Test Coverage Assessment

Gaps:

  • No test for markReady when ensureStarted() rejects
  • No integration test that directly reproduces the original warm-start bug scenario (two disjoint queries with timing-dependent ordering)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants